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SECTION  I 

INTRODUCTION  AND  BACKGROUND 

1.1  NEED  FOR  SQA  MEASUREMENTS 

The  Air  Force  perceives  computer  systems  as  an  increasing 
cost  which  must  be  harnessed  and  controlled.  Computer  systems 
are  rapidly  increasing  in  importance  as  components  of  manage¬ 
ment  and  defense  systems.  The  Air  Force  is  responsible  for 
the  selection,  development  and  maintenance  of  numerous  computer 
systems.  Due  to  a  lack  of  quantifiable  criteria  to  judge  the 
quality  of  a  computer  system  during  its  life  cycle  and  to  other 
causes,  the  acquisition  of  computer  systems  has  resulted  in  cost¬ 
ly  problems  for  the  Air  Force.  These  problems  often  are  not  ob¬ 
served  until  after  the  system  has  been  developed,  and  may  include 
the  following: 

•  Failure  to  meet  user  specifications, 

•  LacK  of  reliability  in  performing  intended  functions, 

•  Inefficient  use  of  computing  resource  and  code, 

•  Failure  to  meet  access  control  requirement, 

e  Difficult  to  use, 

e  Difficult  to  modify  or  upgrade, 

e  Difficult  to  transfer  from  one  system  environment  to 
another , 

e  Unable  to  be  used  in  other  applications,  and 

e  Unable  to  interface  with  other  systems. 

Methods  have  been  developed  to  prevent  or  lessen  the  sever¬ 
ity  of  these  problems.  These  methods  include  training  programs 
for  software  acquisition  personnel,  documentation  of  past 
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experience,  and  procedures  and  tools  to  assure  the  quality  of 
developed  systems.  Guidelines  and  procedures  have  been  provided 
in  a  series  of  software  acquisition  management  guidebooks.  These 
guidebooks  cover  configuration  management,  computer  program 
development  specification,  documentation  requirements,  verifica¬ 
tion,  validation  and  certification,  software  maintenance,  soft¬ 
ware  quality  assurance,  software  cost  estimation  and  measure¬ 
ment,  software  development  and  maintenance  facilities,  and  life 
cycle  events. 

These  guidebooks  provide  qualitative  methods  for  the  assess¬ 
ment  of  software  development.  Quantitative  measures,  or  metrics, 
have  recently  been  developed  as  a  tool  to  quantitatively  measure 
the  quality  of  software.  These  metrics  should  provide  an  addition¬ 
al  means  to  assess  the  quality  of  the  software,  and  to  reduce  or 
prevent  the  problems  listed  above. 

As  a  part  of  its  software  quality  program,  the  Air  Force 
Systems  Command  (AFSC) ,  Electronic  Systems  Division  CESD)  has 
contracted  Systems  Architects,  Inc.  (SAI)  to  develop  a  "Computer 
Systems  Acquisition  Metrics  Handbook".  This  Handbook  will  en¬ 
able  ESD  personnel  to  apply  software  quality  metrics  to  software 
development  efforts.  The  purpose  of  this  theoretical  supplement 
is  to  demonstrate  the  work  done  producing  this  "Computer  Systems 
Acquisition  Metrics  Handbook". 

The  research  in  the  initial  phase  of  this  project  indicated 
that  of  the  work  that  had  been  done  in  the  area  of  software  metrics, 
the  metrics  developed  by  McCall  at  General  Electric  (GE)  (RADC-TR- 
80-109,  Vol.  I  8  II)  were  most  applicable  to  ESD's  environment. 

These  metrics  were  developed  in  the  Department  of  Defense  (DoD) 
environment.  These  metrics  were  adapted  to  form  the  theoretical 
basis  for  the  development  of  this  "Handbook". 


1.2  SOFTWARE  METRICS  LITERATURE  REVIEW 

In  order  to  provide  a  comprehensive  review  of  software  metrics 
and  related  areas  of  research,  SAI  conducted  a  literature  review. 
This  review  met  three  objectives:  (1)  Outline  state-of-the-art 
in  software  metric  efforts;  (2)  Provide  a  scope  of  the  problems 
involved  in  software  development  and  acquisition;  and  (3)  Determine 
what  solutions  have  been  found  to  these  problems  for  incorporation 
into  this  effort  where  relevant. 

The  literature  review  identified  a  number  of  factors  contri¬ 
buting  to  the  poor  development  of  software,  and  some  theories  and 
approaches  developed  to  exert  better  control  and  produce  more  re¬ 
liable  software.  The  areas  of  current  literature  investigated  in¬ 
cluded  metrics,  software  reliability,  software  management,  software 
quality  tools  and  techniques,  and  software  cost  estimation. 

Software  metrics  are  measures  used  to  evaluate  the 
quality  of  software  over  its  life  cycle.  It  is  currently  an  in¬ 
fant  discipline,  and  there  are  a  number  of  conflicting  opinions 
as  to  what  and  how  software  characteristics  should  be  measured. 

Some  of  the  theories  and  methods  that  were  found  for  measuring 
software  characteristics  include  the  following: 

•  Boehm,  Brown,  and  Lipow's  Quality  Characteristics  Tree, 

•  Gilb’s  Software  Metrics, 

•  Halstead’s  Software  Science, 

•  McCall's  Metrics, 

•  McCabe's  Complexity  Measure, 

•  Measures  of  Comprehensibility 
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These  metrics  were  in  the  validation  stage  when  the  early 
efforts  toward  developing  the  "handbook'"  took  place.  The  major 
directions  in  metrics  research  was:  (1)  The  identification  of 
metrics  related  to  a  specific  software  attribute;  and  (2)  The 
use  of  metrics  to  evaluate  programmer  performance  or  human  factors 
involved  in  software  developments.  Areas  of  future  research  are 
software  management  and  automated  measurement.  Considerable  bene¬ 
fits  can  are  are  being  realized  from  this  research. 

Software  reliability  has  been  defined  as  the  degree  to  which 
a  software  system  satisfies  its  requirements ,  and  delivers  usable 
services.  There  are  several  different  types  of  software  relia¬ 
bility  models,  operational  reliability  models,  and  user- oriented 
models.  Although  a  standard  cookbook  approach  for  widespread 
application  of  these  models  cannot  now  be  provided,  software  re¬ 
liability  measurement  have  clearly  progressed  beyond  the  pure 
theory  stage.  The  tools  that  have  been  developed  can  be  useful 
and  valuable  in  software  development  efforts. 

Software  management  includes  the  estimation,  negotiation, 
and  control  of  a  technology  which  is  very  complex  and  which  is 
subject  to  a  high  degree  of  modification.  Typically,  the  fol¬ 
lowing  four  problem  areas  are  encountered: 

(1)  Obtaining  satisfactory  software  requirements, 

(2)  Improving  the  art  of  software  cost  estimating, 

(3)  Achieving  significant  productivity  improvements,  and 

(4)  Maintaining  control  and  visibility  of  software 
developments . 
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Obtaining  satisfactory  software  requirement  specification 
is  the  first  obstacle  to  the  success  of  software  projects.  Eng¬ 
lish  language  statements  are  currently  the  major  form  of  speci¬ 
fication.  Other  forms  of  specification  (including  automated  tools 
and  simulation)  have  been  advocated.  Productivity  can  be  im¬ 
proved  by  providing  software  designers  with  automated  tools. 

The  key  to  effective  control  is  to  break  up  the  development  of 
software  into  a  number  of  small  measurable  steps,  and  then  to 
audit  the  satisfactory  completion  of  those  steps.  Techniques 
necessary  for  controlling  the  development  of  large  software 
based  systems  include:  stepwise  refinement,  requirements 
traceability,  process  design,  incremental  development,  structured 
development,  software  design  language,  unit  development  folders, 
quality  assurance/configuration  management,  and  life  cycle  main¬ 
tenance  . 

There  is  a  wide  range  of  tools  and  techniques  for  improving 
software  quality  over  the  life  cycle.  Specific  tools  and  tech¬ 
niques  have  been  developed  for  software  design,  implementation, 
checkout,  and  maintenance.  Some  of  the  tools  and  techniques  have 
been  applied  to  software  development  projects  economically  and 
with  successful  results,  but  many  are  still  in  Research  and 
Development. 

Software  cost  estimating  is  considered  an  imprecise  art.  At 
the  start  of  this  project  the  best  single  indicator  of  software 
development  cost  was  the  number  of  machine  instructions.  Using 
this  number  has  yielded  high  amounts  of  estimating  errors.  A 
number  of  qualitative  factors  are  needed  to  provide  a  better  pic¬ 
ture  of  the  true  potential  cost.  Two  cost  estimation  models  which 
have  been  developed  include  the  RCA  PRICE-S  and  the  TRW  SCEP. 

Both  of  these  models  have  been  applied  to  a  number  of  software 
development  projects. 


Progress  in  the  advancement  of  software  technology  and  tools 
over  the  past  decade  has  been  rapid,  but  developments  in  software 
have  not  kept  pace  with  the  potential  for  development  provided  by 
hardware  innovations.  The  tools  and  techniques  described  above 
require  further  validation  and  refinement  in  order  to  provide  for 
the  development  of  software  that  is  economical,  reliable,  and 
functional. 

1.3  CONCEPTS  AND  CLASSIFICATION  OF  SOFTWARE  METRICS 
1.3.1  Concepts  of  Software  Metrics 

Software  metrics  is  currently  an  infant  discipline, 
and  there  are  conflicting  opinions  as  to  what  and  how  soft¬ 
ware  characteristics  should  be  measured  (SHNB80)*.  R. 

Rubey  and  R.  Hartwick  first  introduced  the  concept  cf  soft¬ 
ware  metrics  in  1968.  Since  that  time,  the  research  in 
software  metrics  has  grown  to  include  (MCCJ80c)*: 

•  The  use  of  metrics  as  an  aid  in  testing  and 
maintaining  software, 

•  The  psychological  and  programmer  performance 
implications , 

•  The  use  of  metrics  as  software  acceptance 
criteria  in  a  formal  acquisition  environment, 
and 

•  The  use  of  metrics  in  providing  a  quality 
assurance  tool/technique. 

The  major  proposed  methods  for  measuring  computer 
program  quality  are  described  in  the  following  subsections. 
These  proposed  methods  for  software  metrics  include: 

*Key  to  reference  listed  in  Appendix  A. 
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(1)  Boehm,  Brown  and  Lipow's  Quality  Characteristics 
Tree , 

(2)  Gilb's  Software  Metrics, 

(3)  Halstead's  Software  Science, 

(4)  McCall's  Metrics, 

(5)  McCabe's  Complexity  Measure,  and 

(6)  Measures  of  Comprehensibility. 

1.3. 1.1  Boehm,  Brown  and.  Lipow's  Quality 
Characteristic  Tree 

In  determining  what  characteristics  should  be 
measured  in  .order  to  determine  software  quality,  Boehm, 
Brown  and  Lipow  (1978)  defined  a  hierarchical  software 
quality  characteristic  tree.  In  the  quality  character¬ 
istic  tree  (Figure  1-1),  the  arrow  implies  that  the 
quality  to  the  left  of  the  arrow  must  also  include  the 
quality/qualities  to  the  right  of  the  arrow.  Thus,  a 
program  that  is  maintainable  must  also  be  testable, 
understandable,  and  modifiable  (SHNB80) .  The  higher 
levels  of  the  system  reflect  the  uses  of  the  software 
quality  evaluation  in  terms  of: 

(1)  The  current  behavior  of  the  software, 

(2)  The  ease  of  changing  the  software,  and 

(3)  The  ease  of  converting  or  interfacing 
the  software  system  (CURB80b). 

The  lowest  level  characteristics  (right-most 
in  Figure  1-1)  are  "primitives",  or  the  most  basic 
characteristics.  These  lowest  level  characteristics 
may  be  combined  into  the  medium  characteristics  and 
are  recommended  as  software  metrics  for  both  the 
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"primitives"  and  the  higher  level  characteristics. 

The  detailed  and  complex  scheme  proposed  by  Boehm, 

Brown  and  Lipow  is  based  on  practical  experience 
and  is  appealing  to  programmers.  However,  Boehm, 

Brown  and  Lipow  offer  no  clear  cut  demonstration  of 
its  effectiveness,  reliability,  or  applicability  in 
other  environments.  The  lengthy  list  of  issues  acts 
as  a  checklist  for  reviewing  a  program  rather  than 
offering  guidance  in  program  construction.  Such 
checklists  can  be  useful,  but  they  tend  to  grow  long 
and  cumbersome,  match  the  needs  of  a  specific  organiza¬ 
tion,  and  become  language/system  specific  [SHNB80] . 

1.3. 1.2  Gilb's  Software  Metrics 

Gilb  (1977)  presents  a  set  of  basic  software 
metrics,  making  no  claim  as  to  their  completeness.  He 
emphasizes  that  each  software  application  requires  its 
own  concepts  and  that  his  text  is  intended  to  provide 
basic  concepts  on  which  the  user  can  build.  Gilb  builds 
a  strong  and  convincing  case  for  a  precise  measurement 
based  on  the  history  of  the  physical  sciences.  Gilb's 
major  categories  of  metrics  include:  reliability  metrics, 
flexibility  metrics,  structure  metrics,  performance  met¬ 
rics,  resource  metrics,  and  diverse  metrics  [GILB77]. 

Each  of  these  categories  is  broken  down  into  lists  of 
subcategories.  Many  of  Gilb's  metrics  are  difficult  to 
obtain.  Even  where  values  can  be  computed,  there  exists 
no  sense  of  the  range  of  good  values.  The  lack  of  in¬ 
dependence  of  the  metrics  adds  to  the  confusing  complexity 
and  makes  it  difficult  for  programmers  to  predict  the  ef¬ 
fect  of  a  program  change  on  a  group  of  metrics  [SHNB80]. 

1.3. 1.3  Halstead’s  Software  Science 

Halstead  (1977)  approaches  the  human  prepara¬ 
tion  of  computer  programs  by  using  methods  and  princi¬ 
ples  of  classical  experimental  science. 
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This  software  science  is  based  on  counting  the  opera¬ 
tors  and  operands  of  any  program.  This  is  a  difficult 
count  to  make  for  large  programs  uniless  automated.  An 
operator  is  defined  as  the  what-to-do  portion  of  a  pro¬ 
gram  instruction,  and  an  operand  is  a  peice  of  data  upon 
which  an  operation  is  performed.  From  these,  Halstead 
derives  mathematical  formulas  to  determine  estimates  of: 

•  Program  size, 

e  Programming  time, 

•  The  initial  number  of  errors  to  be  expected 

in  a  program,  and 

•  The  number  of  errors  delivered  in  a  program. 

Using  these  formulas,  Halstead  has  derived  a 
program  modularization  scheme.  Programs  modularized  ac¬ 
cording  to  this  scheme,  says  Halstead,  will  be  easy  to 
write,  debug,  comprehend,  and  maintain.  Halstead's  soft¬ 
ware  science  is  appealing,  the  experimental  evidence  is 
convincing,  and  its  psychological  foundations  are  sturdy. 
However,  Halstead's  approach  is  not  complete  since  it 
ignores  specific  issues  such  as  complexity  and  choice  of 
algorithms,  and  general  issues  such  as  portability,  flexi¬ 
bility,  and  efficiency  [SHNB80] .  Halstead's  theory  has 
been  the  subject  of  considerable  research,  and  results 
indicate  that  Halstead's  metrics  are  in  fact  good  indi¬ 
cators  of  the  number  of  bugs  in  a  program,  programming 
time,  and  debugging  time  [CURB80b]. 

1.3. 1.4  McCall's  Metrics 

Research  sponsored  by  the  U.S.  Air  Force  led  to 
McCall's  software  metrics  model.  This  model  contains  a 
comprehensive  hierarchical  definition  of  software  quality 
(see  Figure  1-2).  At  the  highest  level,  quality  factors 
are  defined  that  are  appropriate  for  software  acquisition 
managers  to  use  as  an  aid  in  specifying  quality  objectives 
for  their  software  systems.  These  high  level  factors  are 
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more  software-directed  until  specific  metrics  are  pro¬ 
posed  that  relate  to  the  factors  [CAVJ78] . 

The  application  of  the  framework  for  McCall’s 
metrics  has  three  impacts  on  quality  assurance  activities 
during  large-scale  software  development.  These  impacts 
[MCCJ80v]  are: 

(1)  The  framework  provides  a  mechanism  for 

a  program  manager  to  identify  what  qual¬ 
ities  are  important. 

(2)  The  framework  provides  a  means  of  quan¬ 
titatively  assessing  how  well  the  devel¬ 
opment  is  progressing  relative  to  the  qual¬ 
ity  goals  established. 

(3)  The  framework  provides  for  more  interaction 
by  the  quality  assurance  personnel  through¬ 
out  the  development  effort. 

McCall's  metrics  have  been  documented  in  the 
RADC  Software  Quality  Measurement  Manual.  On-going  work 
involves  automating  the  function  of  data  collection  for 
the  metrics  [MCCJ80] . 

1.3. 1.5  McCabe's  Complexity  Measure 

McCabe  asserts  that  half  of  software  develop¬ 
ment  is  spent  in  testing,  and  most  of  the  money  spent 
on  software  systems  is  used  for  maintaining  the  system. 

Therefore,  what  is  needed  is  a  mathematical  technique 

that  provides  a  quantitative  basis  for  modularization 

and  identifies  software  modules  which  will  be  difficult 

to  test  or  to  maintain  [MCCT76].  * 

McCabe  (1976)  describes  a  complexity  measure  and 
illustrates  its  use  in  managing,  testing,  and  controlling 
program  complexity.  He  defines  this  complexity  measure 
as  the  number  of  paths  through  a  program.  McCabe *s  theory 
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assumes  that  the  complexity  of  a  program  is  not  dependent 
on  its  size  and  that  the  complexity  of  a  program  depends 
only  on  the  structure  of  the  decisions  in  the  program. 
McCabe  says  that  merely  restricting  the  size  of  a  program 
module  does  not  ensure  good  modularization  and  points  out 
that  it  is  possible  for  a  fifty-line  module  to  have  33.5 
million  distinct  control  paths.  His  approach  is  to  mea¬ 
sure  and  control  the  number  of  paths  through  a  program. 

McCabe's  complexity  measure  is  a  helpful  tool 
in  preparing  test  data  and  may  provide  useful  informa¬ 
tion  related  to  program  complexity.  However,  his  metric 
ignores  the  choice  of  algorithms  and  how  data  is  struc¬ 
tured,  and  avoids  important  considerations  such  as  port¬ 
ability,  flexibility,  and  efficiency  [SHNB80] . 

1.3. 1.6  Measures  of  Comprehensibility 

Comprehensibility  is  the  ease  with  which  a  pro¬ 
grammer  can  understand  a  program.  Sheppard,  Borst  and 
Love  (1978)  conducted  an  experiment  to  evaluate  the  ef¬ 
fect  of  structure  on  a  programmer's  understanding  of  a 
computer  program,  and  the  use  of  Halstead’s  and  McCabe's 
metrics  on  the  prediction  of  program  understanding. 

In  the  experiment,  professional  programmers  were 
given  ten  minutes  to  study  a  short  program  and  five  min¬ 
utes  to  reconstruct  an  equivalent  program.  The  experi¬ 
mental  Tesults  indicated  that  the  least  structured  pro¬ 
gram  was  the  most  difficult  to  reconstruct  and  a  partial¬ 
ly  structured  program  was  the  easiest.  McCabe's  complex¬ 
ity  measure  was  found  to  be  a  good  predictor  of  program 
length,  as  was  Halstead's  E  metric  (the  amount  of  effort 
required  to  generate  a  program) . 

Schneiderman  (1978)  has  proposed  that  every 
program  module  should  meet  a  90-10  rule:  "A  competent 
programmer  should  be  able  to  reconstruct  90  percent  of 
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the  program  after  10  minutes  of  study."  This  concept, 
which  is  based  on  several  memorization/reconstruction 
experiments,  needs  testing  and  validation  [SHNB80]. 

1.3. 1.7  Summary 

Two  major  directions  can  be  identified  in  the 
area  of  software  metric  research.  In  one  direction, 
researchers  are  identifying  metrics  related  to  a  specific 
attribute  of  a  software  product's  quality.  The  best 
examples  of  this  type  of  metric  are  the  complexity 
measures  that  have  been  derived.  The  other  direction  has 
been  to  use  metrics  for  the  evaluation  of  programmer  per¬ 
formance  or  the  human  factors  involved  in  software  devel¬ 
opments.  Research  efforts  will  continue  in  the  area  of 
the  psychological  aspects  of  software  development  and  in 
specialized  studies  of  various  metrics  for  each  quality 
factor. 

Other  efforts  will  continue  in  the  area  of  providing 
automated  measurement,  which  is  required  to  make  the 
application  of  metrics  economical  and  consistently  reliable. 
Another  area  that  future  research  will  attach  is  that  of 
software  management,  where  metrics  will  provide  a  feed¬ 
back  mechanism  for  viewing  software  development  as  a 
controlled  process.  Considerable  benefits  can  and  are 
currently  being  realized  from  the  research. 
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SECTION  II 


GE  METRIC  EFFORT 
2.1  SCOPE  OF  GE  METRICS 

Research  sponsored  by  the  U.S.  Air  Force  Electronics  Sys¬ 
tems  Division  (ESD)  and  Rome  Air  Development  Center  (RADC)  led 
to  the  development  of  a  concept  of  software  quality  by  the  Gen¬ 
eral  Electric  Company  (GE) .  One  of  the  objectives  of  this  effort 
was  to  provide  Air  Force  system  acquisition  managers  with  a  mechan¬ 
ism  to  quantitatively  specify  and  measure  the  desired  level  of  qual¬ 
ity  in  a  software  product.  McCall  at  GE  developed  a  system  of 
measures,  or  metrics,  for  the  quantitative  specification  and 
measurement  of  software  quality.  The  concepts,  definitions,  and 
classification  of  these  metrics  are  provided  in  the  subsections 
which  follow. 

2.1.1  Framework 

The  software  metrics  framework  is  a  hierarchical 
structure.  Management -oriented  Quality  Factors  at  the 
highest  level  are  important  to  software  acquisition  managers 
and  used  as  an  aid  in  specifying  quality  objectives  for  their 

software  systems.  At  the  next  level  are  Criteria  which  are 
software-oriented  attributes  used  to  describe  characteris¬ 
tics  of  the  software.  At  the  next  level  are  Metrics  which 
provide  a  measurement  of  the  software  Criteria.  This  hier¬ 
archical  framework  is  presented  in  Figure  II- 1. 

The  measurements  are  to  be  taken  during  the  development 
effort.  These  measurements  are  not  post- implementation  assess¬ 
ments  of  software  quality.  They  are  not  test -like  measure¬ 
ments.  Their  purpose  is  to  provide  an  indication  of  the  pro¬ 
gress  toward  a  desired  level  of  quality  during  the  develop¬ 
ment.  The  Criteria,  established  for  each  software  Quality 
Factor,  represent  attributes  which  can  be  measured  during  the 
software  development.  A  detailed  description  of  each  of  the 
levels  in  this  framework  is  provided  in  the  following  subsections. 
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METRICS 


MANAGEMENT-ORIENTED 
VIEW  OF  PRODUCT  QUALITY 


SOFTWARE-ORIENTED 
ATTRIBUTES  WHICH 
PROVIDE  QUALITY 


QUANTITATIVE  MEASURES 
OF  THOSE  ATTRIBUTES 


t 


FIGURE  II -1 

METRICS  HIERARCHICAL  FRAMEWORK 


2. 1.1.1  Quality  Factors 

Quality  Facotrs  are  management  oriented 
software  qualities  used  to  identify  what  qualities 
are  desired  in  the  software  product  being  developed. 
This  metric  system  uses  the  following  eleven  software 
Quality  Factors. 

(1)  Correctness  -  the  extent  to  which  a 
program  satisfies  its  specifications 
and  fulfills  the  user's  mission 
objectives , 

(2)  Reliability  -  the  extent  to  which  a 
program  can  be  expected  to  perform  its 
intended  function  with  required 
precision, 

(3)  Efficiency  -  the  amount  of  computing 
resources  and  code  required  by  a  pro¬ 
gram  to  perform  a  function, 

(4)  Integrity  -  the, extent  to  which  access 
to  software  or  data  by  unauthorized 
persons  can  be  controlled, 

(5)  Usability  -  the  effort  required  to 
learn,  operate,  prepare  input,  and 
interpret  output  of  a  program, 

(6)  Ma inta inability  -  the  effort  required 
to  locate  and  fix  an  error  in  an  oper¬ 
ational  program, 

(7)  Testability  -  the  effort  required  to 
test  a  program  to  insure  it  performs 
its  intended  function, 
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(8)  Flexibility  -  the  effort  required  to 
modify  an  operational  program, 

(9)  Portability  -  the  effort  required  to 
transfer  a  program  from  one  hardware 
configuration  and/or  software  system 
environment  to  another. 

(10)  Reusability  -  the  extent  to  which  a 
program  can  be  used  in  other  applica¬ 
tions  (related  to  the  packaging  and 
scope  of  the  function  that  programs 
perform) ,  and 

(11)  Interoperability  -  the  effort  required 
to  couple  one  system  with  another. 

These  Factors  provide  a  mechanism  to  spec¬ 
ify  the  basic  attributes  of  a  system  over  its  life 
cycle.  Thus,  if  a  system  is  being  developed  in  an 
environment  where  there  is  a  high  rate  of  technolog¬ 
ical  breakthroughs  in  hardware  design,  portability 
should  be  given  a  primary  significance.  If  the  ex¬ 
pected  life  cycle  of  the  system  is  long,  then  main¬ 
tainability  becomes  a  cost-critical  consideration. 

If  the  system  is  an  experimental  system  where  the 
software  specifications  will  have  a  high  rate  of 
change,  flexibility  in  the  software  product  is  highly 
desirable.  If  the  functions  of  the  system  are  ex¬ 
pected  to  be  required  for  a  long  time,  while  the 
system  itself  may  change  considerably  from  time  to 
time,  then  reusability  is  highly  significant  for  those 
modules  which  implement  the  major  functions  of  the 
system.  The  quality  of  interoperability  becomes  ex¬ 
tremely  important  for  systems  required  to  interface 
with  others  via  communications  networks. 
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2. 1.1. 2  Criteria 

Software  Criteria  are  attributes  of  the 
software  and  further  define  the  hierarchical  struc¬ 
ture  of  GE's  metrics.  Each  Factor  is  defined  by  a 
set  of  Criteria;  Criteria  which  affect  more  than  one 
Factor  help  to  describe  the  relationships  between 
Factors.  The  software  Criteria  are  listed  and  de¬ 
fined  in  Table  II -A.  The  relationship  between  Cri¬ 
teria  and  Factors  is  shown  in  Table  II-B. 

2. 1. 1. 3  Metrics 

The  actual  measurement  of  software  quality 
is  accomplished  by  applying  Software  Metrics  to  the 
documentation  and  source  code  produced  during  a  soft¬ 
ware  development  effort.  To  determine  the  value  of 
a  metric  questions  are  asked  that  are  answerable 
numerically  or  by  Yes  or  No  response.  The  Yes  or 
No  responses  are  translated  to  1  and  0  respectively 
so  that  in  effect  all  answers  are  numerical.  Metrics 
are  established  in  a  one  to  one  relationship  with 
criteria  to  provide  a  measure  of  the  software  Criteria. 
Which  in  turn  combine  through  algorithms  to  Quantify 
the  Quality  Factor. 

The  metrics  are  organized  using  five  phases 
of  software  development:  (1)  Requirements  Analysis 
(2)  Preliminary  Design,  (3)  Detail  Design,  (4)  Imple¬ 
mentation,  and  (5)  Test  and  Integration.  See  Figure 
1 1  -  2  . 
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CRITERIA 


DEFINITION 


Traceability  Those  attributes  of  the  software  that  pro¬ 

vide  a  thread  from  the  requirements  to  the 
implementation,  with  respect  to  the  speci¬ 
fic  development  and  operational  environment. 


Completeness  Those  attributes  of  the  software  that  pro¬ 

vide  full  implementation  of  the  functions 
required. 


Consistency  Those  attributes  of  the  software  that  pro¬ 

vide  uniform  design  and  implementation 
techniques  and  notation. 


Accuracy  Those  attributes  of  the  software  that  pro¬ 

vide  the  required  precision  in  calculations 
and  outputs. 


Error  Tolerance  Those  attributes  of  the  software  that  pro¬ 
vide  continuity  of  operation  under  non- 
nominal  conditions. 


Simplicity  Those  attributes  of  the  software  that  pro¬ 

vide  implementation  of  functions  in  the 
most  understandable  manner.  (Usually 
avoidance  of  practices  which  increase 
complexity. ) 


Modularity  Those  attributes  of  the  software  that  pro¬ 

vide  a  structure  of  highly  independent 
modules . 


Generality  Those  attributes  of  the  software  that  pro¬ 

vide  breadth  to  the  functions  performed. 
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SOFTWARE  CRITERIA 


CRITERIA 


DEFINITION 


Expandability 

Those  attributes  of  the  software  that  pro¬ 
vide  for  expansion  of  data  storage  require¬ 
ments  or  computational  functions. 

Instrumentation 

Those  attributes  of  the  software  that  pro¬ 
vide  for  the  measurement  of  usage  or  iden¬ 
tification  of  errors. 

Self- 

Descr iptiveness 

Those  attributes  of  the  software  that 
provide  explanation  of  the  implementation 
of  a  function. 

Execution 

Efficiency 

Those  attributes  of  the  software  that 
provide  for  minimum  processing  time. 

Storage 

Efficiency 

Those  attributes  of  the  software  that 
provide  for  minimum  storage  requirements 
during  operation. 

Access  Control 

Those  attributes  of  the  software  that  pro¬ 
vide  for  control  of  the  access  of  software 
and  data. 

Access  Audit 

Those  attributes  of  the  software  that  pro¬ 
vide  for  an  audit  of  the  access  of  software 
and  data. 

Operability 

Those  attributes  of  the  software  that 
determine  operation  and  procedures  con¬ 
cerned  with  the  operation  of  the  software. 
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SOFTWARE  CRITERIA 


CRITERIA 

DEFINITION 

Training 

Those  attributes  of  the  software  that  pro¬ 
vide  transition  from  current  operation  or 
initial  familiarization. 

Communicativeness 

Those  attributes  of  the  software  that  pro¬ 
vide  useful  inputs  and  outputs  which  can  be 
assimilated. 

Software  System 
Independence 

Those  attributes  of  the  software  that 
determine  its  dependency  on  the  software 
environment  (operating  systems,  utilities, 
input/output  routines,  etc.). 

Machine 

Independence 

• 

Those  attributes  of  the  software  that 
determine  its  dependency  on  the  hardware 
system. 

Communications 

Commonality 

Those  attributes  of  the  software  that 
provide  the  use  of  standard  protocols  and 
interface  routines. 

Data  Commonality 

Those  attributes  of  the  software  that  pro¬ 
vide  the  use  of  standard  data  represen¬ 
tations  . 

Conciseness 

Those  attributes  of  the  software  that  pro¬ 
vide  for  implementation  of  a  function  with 
a  minimum  amount  of  code. 

TABLE  1 1 -A  (continued! 
SOFTWARE  CRITERIA 


FACTOR 


CRITERIA 


Operability 

Training 

Communicativeness 

Access  Control 

Access  Audit 

Storage  Efficiency 

Execution  Efficiency 

Traceability 

Completeness 

Accuracy 

Error  Tolerance 

Consistency 

Simplicity 

Conciseness 

Instrumentation 

Expandability 

Generality 

Self-Descriptiveness 

Modularity 

Machine  Independence 

Software  System  Independence 

Communications  Commonality 

Data  Commonality 


TABLE  II-B 

RELATIONSHIP  OF  FACTORS  AND  CRITERIA 


The  metrics  can  be  applied  either  at  the  system  level  or 
subsystem  level  if  the  subsystem  is  viewed  as  the  "System”. 
At  the  system  level,  the  metrics  can  be  utilized  to  obtain 
an  overall  measure  of  how  the  system  is  progressing  with 
respect  to  a- particular  Quality  Factor.  At  the  subsystem 
level,  the  metrics  can  be  used  to  identify  problems  in 
a  particular  subsystem  so  that  corrective  actions  or  an 
emphasis  can  be  applied  to  that  subsystem.  Appendix  B  of 
GE's  Software  Quality  Metrics  Enhancements  (RADC-TR-80- 
109)  provide  a  description  and  explanation  of  the  use 
of  GE's  metrics. 

2.2  GE  PROCEDURE  FOR  APPLYING  METRICS 

The  procedure  used  to  apply  GE's  metrics  involves  three 
steps:  (1)  Identify  software  quality  requirements,  (2)  Apply 

software  quality  measurements,  and  (3)  Assess  the  quality  of  the 
software  product.  The  procedures  involved  in  these  steps  are 
described  in  the  following  subsections. 

2.2.1  Identify  Software  Quality  Requirements 

Activities  in  identifying  software  quality  require¬ 
ments  include  identifying  important  Quality  Factors,  iden¬ 
tifying  critical  software  attributes,  and  establishing 
quantifiable  goals.  To  identify  software  quality  require¬ 
ments,  a  survey  form  is  used  to  solicit  responses  from  the 
system's  decision  makers  (acquisition  manager,  user/customer, 
development  manager,  and  quality  assurance  manager). 

This  survey  form,  shown  in  Figure  II-3,  is  used  to 
gather  information  with  respect  to  the  basic  characteristics 
of  the  application.  This  identifies  the  rank  of  Quality 
Factors,  and  documents  the  rationale  for  the  decisions  made 
in  selecting  the  Quality  Factors. 
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FIGURE  1 1 -2 


TIMING  OF  THE  APPLICATION  OF  THE  METRIC  WORKSHEETS 
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1.  The  11  quality  factors  listed  below  have  boon  Isolated  froai  tho  cur¬ 
rent  11 teraturo.  They  are  not  want  to  be  exhaustive,  but  to  reflect 
what  Is  currently  thought  to  be  Important.  Please  Indicate  whether 
you  consider  each  factor  to  be  Very  Important  (VI),  Important  (I), 
Somarfiat  Important  (SI),  or  Not  Important  (NI)  as  design  goals  in  the 
system  you  are  currently  working  on. 
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2.  What  type(s)  of  application  are  you  currently  Involved  In? 


3.  Are  you  currently  In: 

___________  1 .  Development  phase 

________  2.  Operatlons/Malntenance  phase 

4.  Please  Indicate  the  title  which  most  closely  describes  your  position: 

.  1 .  Program  fenagor 

_______  2.  Technical  Consultant 

_______  3.  Systems  Analyst 

______  4.  Other  (pleese  specify) 
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SOFTWARE  QUALITY  REQUIREMENTS  SURVEY  FORM 


The  degree  of  Relationship  between  software  Quality  Factors 
is  shown  in  Figure  1 1 - 4 .  This  is  used  to  "trade-off"  Quality 
Factors  selection  so  that  Quality  Factors  with  a  low  degree 
of  relationships  are  not  expected  to  exist  simultaneously 
within  a  system. 

The  next  level  of  identifying  the  quality  measurements 
is  to  proceed  from  the  management- oriented  Quality  Factors 
to  the  software-oriented  Criteria.  The  Criteria  are  re¬ 
lated  to  the  various  factors  by  definition  and  provide  a  more 
detailed  specification  of  the  quality  requirements. 

After  the  critical  quality  factors  have  been  identi¬ 
fied,  specific  performance  levels  or  ratings  required  for 
each  factor  should  be  specified.  The  specific  metrics  which 
will  be  applied  to  the  various  software  products  produced 
during  the  development  should  be  specified,  and  specific 
minimum  values  for  particular  metrics  may  be  specified  in 
addition  to  the  ratings. 

2.2.2  Apply  Software  Quality  Measurements 

The  vehicle  for  applying  the  software  quality  measure¬ 
ments  are  the  metric  worksheets  contained  in  GE's  Software 
Quality  Metrics  Manual  (RADC-TR-80-109) .  The  metric  work¬ 
sheets  are  applied  to  the  available  system  documentation  or 
source  code  and  the  measurements  are  translated  into  metric 
scores.  The  application  of  the  metric  worksheets  follow  the 
phased  development  of  the  software.  The  timing  of  the  ap¬ 
plication  of  the  metric  worksheets  is  shown  in  Figure  1 1 - 2 . 

2.3  INTERPRETATION  OF  GE  METRICS 

The  benefits  of  applying  the  software  quality  metrics  are 
realized  when  the  information  gathered  from  the  application  of  the 
metric  worksheets  is  analyzed.  There  are  three  levels  at  which 
analyses  can  be  performed:  (1)  Inspector's  Assessment,  (2)  Sensi¬ 
tivity  Analysis,  and  (3)  Use  of  Normalization  Function.  In  the 
Inspector's  Assessment,  an  inspector,  using  the  worksheets,  asks 
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the  same  questions  and  takes  the  same  counts  for  each  module's 
source  code  or  design  document  that  is  reviewed.  Based  on  this 
consistent  evaluation,  a  subjective  comparison  of  the  products 
can  be  made. 

The  Sensitivity  Analysis  uses  the  results  of  the  calcula¬ 
tions  from  the  metric  worksheets  to  form  a  matrix  of  measurements. 
The  matrix  represents  a  profile  of  all  the  modules  in  the  system 
with  respect  to  a  number  of  characteristics  measured  by  the 
metrics,  and  allows  a  number  of  analyses  to  be  performed.  The 
last  level  of  quality  assessment  involves  using  normalization 
functions  to  predict  the  quality  of  the  software  in  quantitative 
terms . 

The  majority  of  the  coefficients  required  to  perform  the 
sensitivity  analysis  are  not  available  at  this  time  due  to  their 
empirical  nature.  More  research  and  development  must  be  done  in 
this  area  to  provide  full  information  at  all  three  levels  of 
interpretation. 
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SECTION  III 


METHODOLOGY  FOR  TRANSFORMING  THE  GE  METRICS 
INTO  THE  SOFTWARE  METRICS  HANDBOOK 

3.1  OBJECTIVES 

The  objectives  of  this  project  were  to  develop  a  standard  set 
of  procedures  that  quantitatively  specify  and  measure  the  quality 
of  a  software  system  during  its  life  cycle.  The  procedures  were 
written  so  that  they  could  be  learned  during  a  one  week  training 
program  by  students  with-limited  software  system  development  knowledge. 
These  procedures  are  documented  in  the  form  of  a  "Computer  Systems 

Acquisition  Metrics  Handbook"  (sometimes  referred  to  herein  as  the 
"Handbook") . 

The  development  methodology  employed  by  SAI  in  achieving 
these  objectives  can  be  described  at  its  highest  level  in  five 
steps . 

Step  1.  Gain  complete  knowledge  of  the  computer  metrics 

technology  developed  by  General  Electric  in  April 
1980  (RADC-TR-80-109,  Vol.  I  8  II) . 

Step  2.  Employ  a  stepwise  refinement  to  decompose  the  GE 
metric  system  to  successively  increasing  levels 
of  detail  until  all  fundamental  units,  the  "Data 
Elements",  are  described  and  understood. 

Step  3.  Evaluate  each  "Data  Element"  to  identify  those 
suitable  for  the  "Handbook". 

Step  4.  Rebuild  the  system  from  the  bottom  up  using  only 
those  "Data  Elements"  selected  in  Step  3. 

Step  5.  Present  the  metric  system  in  the  form  of  a 
"Practical  Handbook". 

As  illustrated  in  Figure  III-l,  the  "Framework  of  the  Metrics 
Handbook"  has  the  following  five  components:  (1)  General  Instruc¬ 
tions,  (2)  Quality  Factor  Selection  Instructions,  (3)  Data 
Element  Dictionary,  (4)  Quality  Factor  Modules  (eleven),  and 
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(5)  Sets  of  Data  Collection,  Metrics,  Evaluation  Worksheets, 

Criteria  for  metric  and  Factor  level  organized  according  to  the 
life  cycle  model.  The  procedure  for  applying  the  "Handbook"  is 
hierarchical  and  can  be  described  at  its  highest  level  in  four  steps; 

Step  1.  Decide  whether  the  handbook  is  suitable  for  the 
system  under  development  based  on  the  "General 
Instructions". 

Step  2.  Select  the  Quality  Factors  relevant  to  .the 

system  based  on  the  "Quality  Factor  Selection 
Instructions". 

Step  3.  Obtain  only  those  Quality  Factor  Modules  that 
correspond  to  the  Quality  Factors  selected  in 
Step  2 . 

Step  4.  Apply  the  worksheets  repeatedly  over  the  system 
life  cycle  as  described  in  the  "Quality  Factor 
Module  Instructions". 

This  framework  of  procedure  provides  a  software  quality  measure¬ 
ment  system  which  is  quantitative  in  value,  yet  simple  enough 
to  be  learned  in  one  week  by  a  student  with  limited  software 
development  background. 

A  sample  worksheet  from  the  Handbook  is  included  on  the 
following  page.  In  the  upper  right  hand  corner  of  each  work¬ 
sheet  is  the  Form  Code.  Each  worksheet  is  assigned  a  Form 
Code  according  to  the  Form  Code  Key  below  in  Table  II I -A. 

When  the  worksheets  are  organized  according  to  Form  Code , 
the  eleven  "Quality  Factor  Modules"  are  formed. 
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The  following  development  methodology  described  in  the  balance 
of  this  section  expands  on  the  evaluation  of  each  "Data  Element" 
in  the  GE  metric  system  to  identify  those  suitable  for  the  "Computer 
System  Acquisition  Metrics  Handbook".  Those  selected  are  included 
in  the  "Worksheets"  of  the  eleven  "Quality  Factor  Modules". 

3.2  EVALUATION  CRITERIA  FOR  DEVELOPING  THE  COMPUTER  SYSTEM 

The  SAI  technical  staff  established  evaluation  criteria  for 
selecting  Data  Elements  to  be  used  in  the  handbook.  This  was 
done  so  only  suitable  Data  Elements  would  be  included  while  ob¬ 
scure  or  difficult  Data  Elements  would  be  excluded.  The  three 
criteria  chosen  for  this  purpose  were  (1)  Period,  (2)  Training, 
and  (3)  Importance.  The  degree  of  significance  of  each  data  ele¬ 
ment  was  evaluated  against  each  of  the  three  criteria  and  the 
resulting  score  plotted  in  a  three-dimensional  space  with  the 
three  criteria  as  unit  vectors.  Figure  I II - 2  illustrates  this 
concept.  The  three  criteria  were  selected  for  their  orthogonality 
and  significance  relative  to  the  objective  of  the  "Handbook".  A 
three-dimensional  space  was  chosen  as  a  model  to  preserve  the  con¬ 
cept  of  orthogonality. 
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3.2.1  Period 

The  evaluation  criterion  Period  is  based  on  the  amount 
of  time  in  minutes  it  takes  for  the  person  making  the  measure¬ 
ments  to  acquire  information  asked  for  in  the  Data  Element  . 

3.2.2  Importance 

The  evaluation  criterion  Importance  is  based  on  the 
number  of  times  a  Data  Element  occurs  in  the  total  number  of 
measurements  and  how  effective  it  is  in  the  related  Petrie 
algorithims. 
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The  following  development  methodology  described  in  the  balance 
of  this  section  expands  on  the  evaluation  of  each  "Data  Element" 
in  the  GE  metric  system  to  identify  those  suitable  for  the  "Computer 
System  Acquisition  Metrics  Handbook".  Those  selected  are  included 
in  the  "Worksheets"  of  the  eleven  "Quality  Factor  Modules". 

3.2  EVALUATION  CRITERIA  FOR  DEVELOPING  THE  COMPUTER  SYSTEM 

ACWrSTTTftKrHANPBOOK - — - 

The  SAI  technical  staff  established  evaluation  criteria  for 
selecting  Data  Elements  to  be  used  in  the  handbook.  This  was 
done  so  only  suitable  Data  Elements  would  be  included  while  ob¬ 
scure  or  difficult  Data  Elements  would  be  excluded.  The  three 
criteria  chosen  for  this  purpose  were  (1)  Period,  (2)  Training, 
and  (3)  Importance.  The  degree  of  significance  of  each  data  ele¬ 
ment  was  evaluated  against  each  of  the  three  criteria  and  the 
resulting  score  plotted  in  a  three-dimensional  space  with  the 
three  criteria  as  unit  vectors.  Figure  1 1 1 - 2  illustrates  this 
concept.  The  three  criteria  were  selected  for  their  orthogonality 
and  significance  relative  to  the  objective  of  the  "Handbook".  A 
three-dimensional  space  was  chosen  as  a  model  to  preserve  the  con¬ 
cept  of  orthogonality. 
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The  evaluation  criterion  Period  is  based  on  the  amount 
of  time  in  minutes  it  takes  for  the  person  making  the  measure¬ 
ments  to  acquire  information  asked  for  in  the  Data  Element  . 

3.2.2  Importance 

The  evaluation  criterion  Importance  is  based  on  the 
number  of  times  a  Data  Element  occurs  in  the  total  number  of 
measurements  and  how  effective  it  is  in  the  related  metric 
algorithims. 
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3.2.3  Training 

The  evaluation  criterion  Training  is  based  on  the 
amount  of  time  it  takes  for  a  person  to  learn  and  understand 
a  Data  Element.  This  time  estimate  is  based  on  the  amount 
of  software  experience  anticipated  in  the  target  audience. 

3.3  THE  DATA  ELEMENTS  FROM  THE  GE  WORKSHEETS 

Data  Elements  are  a  set  of  very  specific  questions  about 
system/software  characteristics  over  the  development  life  cycle. 
These  questions  are  the  main  initial  materials  for  the  "Handbook: 
development.  SAI  identified  all  the  different  Data  Elements  from 
the  four  GE  Metrics  worksheets.  The  Data  Elements  can  be  found  in 
their  original  form  on  Metric  Worksheets  1,  2a,  2b,  and  3,  pages  38 
through  50  of  RADC-TR-80-109 ,  Vol.  II,  Final  Technical  Report, 

April  1980,  Software  Quality  Measured  Manual. 

3.4  METHODOLOGY  FOR  EVALUATING  DATA  ELEMENTS  SUITABLE  FOR  THE 
HANDBOOK 

Once  all  the  Data  Elements  were  identified,  the  following 
evaluation  was  applied.  Each  Data  Element  was  measured  according 
to  the  three  evaluation  criteria  described  in  Section  3.2,  (Per¬ 
iod,  Importance,  and  Training).  Each  Data  Element  received  a 
Period  score,  an  Importance  score,  and  a  Training  score.  Each 
score  ranges  from  one  to  five  for  each  Data  Element,  and  was 
represented  by  a  point  in  the  three-dimensional  space  of  Figure 
1 1 1 - 3 .  Sections  3.4.1  through  3.4.3  describe  the  methodology  that 
was  employed  to  assign  Period,  Importance,  and  Training  scores  for 
the  Data  Elements. 

3.4.1  Period  Score 

For  the  evaluation  criterion  Period,  a  Data  Element  was 
scored  according  to  the  approximate  time  in  minutes  it  would 
take  someone  to  acquire  information  asked  for  by  that  Data 
Element.  The  score  has  a  range  of  1  to  5,  each  score  corres¬ 
ponding  to  a  period  of  time  in  minutes.  The  less  time  that  is 
involved,  the  higher  the  score.  Figure  1 1 1  -  4  illustrates 
the  Period  algorithm. 
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3.4.2  Importance  Score 

For  the  evaluation  criterion  Importance,  a  Data  Ele¬ 
ment  was  scored  according  to  (1)  the  number  of  occurrences 
in  the  metric  system,  and  (2)  its  effectiveness. 
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within  the  related  metric  algorithm.  The  Importance  score  is 
acquired  by  adding  the  Occurrence  and  Effective  scores.  Figure 
III- 3  illustrates  the  Importance  Algorithm. 
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3.4.3  Training  Score 

For  the  evaluation  criterion,  Training,  a  Data 
Element  was  scored  according  to:  (1)  The  approximate 
time  in  minutes  it  would  take  someone  to  understand  a 
Data  Element  (Explanation  Time} ;  and  (2)  The  amount  of 
related  experience  that  person  had  (Experience) .  A 
short  explanation  time  scored  high  and  a  low  experience 
scored  high.  The  Training  score  is  obtained  by 
adding  the  Explanation  Time  and  Experience  scores. 

Figure  III-  6  illustrates  the  Training  Algorithm. 

3.5  SELECTING  DATA  ELEMENTS  SUITABLE  FOR  THE  HANDBOOK 

After  assigning  a  score  in  each  of  the  three  dimensions 
for  each  Data  Element,  the  next  step  was  to  select  a  method 
of  combining  the  three  scores  to  produce  a  compos it  score  for 
each  Data  Element.  The  purpose  was  to  give  each  Data  Element 
a  score  based  on  all  three  evaluation  criteria.  This  was 
applied  to  all  Data  Elements.  Once  all  the  Data  Elements 
mere  represented  by  one  score,  a  score  distribution  analysis 
was  conducted.  This  classified  the  Data  Elements  into  three 
categories:  (1)  Keep;  (2)  Reserve;  and  (3)  Drop.  The  following 

subsections  describe  this  whole  process  in  detail. 

3.5.1  Candidate  Selection  Methodologies 

SAI  considered  three  composit  scoring  methods  for 
the  Data  Elements:  (1)  Sum;  (2)  Vector  Length;  and 

(3)  Product.  The  Data  Element  score  in  the  Sum  method 
is  derived  from  the  sum  of  the  Data  Element's  three 
evaluation  criteria  scores.  The  Data  Element  score  in 
the  Vector  Length  method  is  derived  from  the  square  root 
of  the  sum  of  the  squared  criteria  scores.  The  Data  Element 
score  in  the  Product  method  is  derived  from  the  product  of 
the  criteria  scores. 


Ill -11 


3.5.2  Comparison -of  Selection  Methodologies 

SAI  tested  each  candidate  selection  methodology 
on  all  the  Data  Elements  to  determine  the  most  efficient 
one  for  the  Data  Element  selection  process.  Figure  III- 7 
illustrates  examples  of  the  three  methods  on  one  Data 
Element  that  has  a  Period  score  of  5,  an  Importance 
score  of  4,  and  a  Training  score  of  3. 

3.5.3  Best  Selection  Methodology 

SAI  chose  the  Sum  method  as  the  best  selection 
methodology  because  the  distribution  of  scores  derived 
from  it  was  smooth  and  clear  compared  to  the  other  methods. 
Using  the  sum  method  as  a  base  of  scores  for  the  Data 
Elements,  SAI  classified  each  of  the  Data  Elements  into 
a  population  of  three  classes:  (1)  Keep;  (2)  Reserve; 
and  (3)  Drop.  The  mean  and  standard  deviation  of  the 
Data  Element  score  distribution  establishes  the  Keep, 

Reserve,  and  Drop  categories.  Figure  III-8  illustrates 
the  distribution.  Twenty  Data  Elements  with  low  scores 
were  dropped;  33  with  higher  scores  were  put  on  Reserve, 
while  the  rest  were  kept. 

3.5.4  Comparison  of  GE  and  SAI  Data  Element  Evaluations 

The  next  step  was  to  compare  SAI  and  GE  scoring 
criteria  to  see  if  the  ratings  tl»ey  gave  for  the  Data 
Elements  were  significantly  different.  Figures  III-9  and 
III-10  show  the  scoring  criteria  utilized  by  GE  and  SAI. 
Notice  that  the  SAI  criterion  time  is  equivalent  to  GE's 
Criterion  Effort  and  SAI's  Training  Requirement  is  equivalent 
to  GE's  Skill  Level. 
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Figure  II I- 11  shows  the  comparison  between  the  two 
scoring  methodologies.  GE's  Skill  Level  and  Effort  scores 
range  from  1  (lowest)  to  5  (highest)  while  SAI's  scores 
are  the  exact  opposite.  Both  GE  and  SAI  score  criteria 
were  used  to  measure  the  Data  Elements.  Table  III-B 
shows  a  score  distribution,  average  score  and  results  of 
applying  Skill  Level  measurements  to  the  Data  Elements. 

Table  III-C  shows  the  same  type  of  analysis  but  with  the 
Effort  measurement,  while  Table  III-  D shows  both  criteria 
combined.  The  figures  show  that  there  was  a  slight  difference 
in  the  scoring  results  but  not  a  significant  one. 


3.5.S  Impact  of  Reducing  Number  of  Data  Elements  on 
Quality  factors' 


It  was  important  to  assess  what  "dropping"  data 
would  do  at  higher  levels.  Would  dropping  a  set  of  Data 
Elements  be  equivalent  to  eliminating  a  Quality  Factor? 


As  Table  III-E  shows,  dropping  20  Data  Elements 
resulted  in  the  elimination  of  two  metrics  from  the  system. 
Clearly,  there  was  no  significant  impact  on  any  of  the  11 
Quality  Factors. 

3.5.6  Categories  of  Data  Elements 

SAI  classified  146  GE  51  ata  Elements  before  the 
selection  process  described  above.  Of  these,  93  fell  into 
the  Keep  Category,  33  fell  into  the  Reserve  Category,  and 
20  fell  into  the  Drop  Category.  Tables  III-F  to  III-H 
list  the  Data  Elements  according  to  category  along  with 
the  score  under  all  these  selection  methodologies.  The 
Data  Element  listed  by  the  SAI  sequence  code  (numerical 
value)  and  the  GE  code,  as  defined  in  RADC-TR-80-109,  are 
in  the  left  columns  of  the  tables.  The  columns  to  the 
right  show  the  scores  of  the  data  elements  derived  from 
the  Sum,  Vector  Length,  and  Product  Scoring  methodologies. 
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3.6  MAPPING  OF  G.E.  METRICS  INTO  SOFTWARE  METRICS  USED  IN 

THIS  H5NSBTOK - 

All  126  data  elements  in  the  Keep  and  Reserve  categories 
were  selected  for  the  Metrics  handbook..  The  ones  in  the  Drop 
category  were  not  included.  Tables  III-F  and  III-G  list  the 
data  elements  that  were  suitable  for  the  "Handbook".  They 
were  incorporated  into  the  worksheets  according  to  the 
Framework  described  in  Subsection  3.1. 

The  bottom-up  approach,  going  from  data  element  to  metric 
to  criteria  and  finally  Quality  Factor  according  to  the 
original  framework  was  observed  and  produces  the  modules  of 
this  handbook. 
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PILOT  APPLICATION  OF  SOFTWARE  METRICS  HANDBOOK 

4.1  TRAINING  PHASE  OF  SOFTWARE  METRICS  HANDBOOK 

A  training  course  for  ESD/TOEE  personnel  on  the  metrics 
concept  and  use  of  the  "handbook"  procedures  was  held  at  ESD/TOEE 
during  the  last  phase  of  the  contract.  The  purpose  of  this 
training  course  was  to  demonstrate  that  the  Software  Metrics 
"handbook"  can  easily  be  learned  by  inexperienced  personnel. 

The  materials  developed  for  this  course  and  modified  as  a  result 
of  this  training  phase  include:  (1)  Software  Metrics  Handbook; 

(2)  Quality  Factor  Module  Instructions;  (3)  Quality  Factor 
Modules;  (4)  Data  Element  Dictionary;  (5)  Instructor's  Guide; 

(6)  Course  Outline;  and  related  training  material  and  transparencies 
These  materials  will  allow  ESD/TOEE  to  instruct  their  personnel 
in  the  application  of  the  Software  Metrics  Handbook. 

4.2  APPLICATION  OF  SOFTWARE  METRICS  HANDBOOK  TO  A  C5  SYSTEM 

The  Software  Metrics  Handbook  was  applied  to  an  actual  C3 
system  by  ESD/TOEE  personnel  working  with  SAI  personnel.  The 
system,  provided  by  ESD,  was  the  RADAR  Prediction  System  (RAPS) . 

The  documentation  for  this  system  consisted  of  the  System 
Specification,  Development  Specifications  for  various  subsystems, 
Product  Specifications  for  various  subsystems  and  Source  Code. 

The  Metrics  were  applied  on  six  separate  occasions,  the  purpose 
being  two-fold:  To  validate  the  applicability  of  the  Software 
Metrics  Handbook  and  to  improve  the  Handbook  components .  This 
application  resulted  in  the  "fine  tuning"  of  the  Software  Metrics 
Handbook. 

4.3  ISSUES  AND  RESOLUTIONS 

During  the  application  of  the  Software  Metrics  Handbook 
several  issues  were  raised  regarding  the  Handbook  components. 

These  issues  and  resolutions  are  as  follows: 
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Issue  #1:  The  Quality  Factor  Selection  method  was  too 
proceduralized. 

Resolution  #1:  The  General  Instructions  were  modified  to 
deproceduralize  the  selection  method. 

Issue  #2:  Certain  information  required  by  the  Quality  Factor 
selection  forms  was  deemed  unnecessary. 

Resolution  #2:  This  information  requirement  was  removed 

from  the  Quality  Factor  Selection  Survey  Form. 

Issue  #3:  Information  should  be  included  regarding  cost,  time, 
and  scoring  f  Software  Metrics  during  the  tradeoff 
process . 

Resolution  #3:  These  issues  are  addressed  in  the  training 
materials  prepared  for  ESD  to  teach  the  use 
of  the  Handbook. 

Issue  #4:  The  flow  of  information  between  the  General 

Instructions  and  the  Module  Instructions  should  be 
clarified. 

Resolution  #4:  This  clarification  was  achieved  by  modifying 
Section  IV  of  the  General  Instructions  - 
"How  The  Handbook  Works". 

Issue  S:  The  definition  of  "Module"  must  be  clarified  and  the 
concepts  of  and  applications  of  system  level  as 
opposed  to  module  level  should  be  discussed. 

Resolution  #5:  These  clarifications  and  discussions  are 

included  in  the  General  Instructions.  Section 
I  -  "Software  Metrics". 

Issue  #6:  More  information  must  be  supplied  regarding  the 
correct  sequence  for  applying  the  worksheets. 

Resolution  #6:  This  information  is  provided  in  the  General 

Instructions ,  Section  I  *  "Software  Metrics". 
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Issue  #7: 

Resolution 

Issue  #8: 

Resolution 

Issue  #9: 

Resolution 

Issue  #10: 

Resolution 

Issue  #11: 

Resolution 

Issue  #12: 

Resolution 


The  section  labeled,  "Source"  on  the  worksheets 
should  be  filled  in  by  the  person  using  it  with 
the  name  of  the  system. 

#7:  The  worksheet  format  was  changed  and  instructions 
included  in  the  General  Instructions. 

The  concepts  of  granularity  and  subjectivity 
regarding  the  data  elements  need  to  be  addressed 
in  the  Theoretical  Supplement. 

#8:  Included  in  Section  5.2  of  the  Theoretical 

Supplement. 

Blocks  marked  "System  Level",  "Module  Level",  and 
"Subsystem  Level"  should  be  added  to  the  worksheets. 

#9:  The  worksheets  were  modified  to  reflect  this 
change . 

Products  may  not  map  uniquely  into  a  single  life 
cycle  phase.  As  a  result,  information  referenced 
in  preceding  phases  may  be  required  to  apply 
Software  Metrics  at  a  current  phase. 

#10:  This  issue  is  resolved  in  the  General 
Instructions ♦ 

Emphasis  should  be  placed  on  the  content  of 
products  and  not  the  name  or  type  of  products 
when  applying  the  Software  Metrics  to  system 
documentation. 

#11:  This  issue  is  resolved  in  the  General 
Instructions . 

The  Metrics  "Training  Checklist"  in  the  Preliminary 
Design  phase  of  the  Quality  Factor  Module  Usability 
should  be  included  in  the  Requirements  Analysis 
phase  and  deleted  from  the  Prelimanary  Design  phase. 

#12:  The  worksheets  in  the  Usability.  Module  were 
modified. 
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Issue  #13: 

Resolution 

Issue  #14: 

Resolution 

Issue  #15: 

Resolution 

Issue  #16: 

Resolution 

Issue  #17: 

Resolution 


There  is  a  lack  of  information  regarding  standards 
and  conventions  etc.  in  the  Detail  Design  Phase. 

#13:  This  issue  was  resolved  by  including  the 
Computer  Program  Development  Plan  (CPDP) 
in  the  product  tables  for  the  eleven  Quality 
Factor  Modules,  and  including  this  information 
in  the  General  Instructions,  Section  II  - 
"Life  Cycle  Considerations". 

The  Quality  Factor  Modules  may  not  be  easily 
reproduced  for  use  by  ESD  personnel. 

#14:  The  masters  will  be  delivered  to  ESD  unbound 
to  facilitate  making  copies. 

All  modules  may  not  be  applicable  to  all  Computer 
Program  Configuration  Items  (CPCI)  or  not  applicable 
due  to  system  characteristics. 

#15:  The  General  Instructions  were  expanded  to 
include  this  information. 

The  Interpretation  Worksection  should  be  changed 
to  evaluate  the  reviewable  products  on  a  1  to  10 
scale  subjectively. 

#16:  The  worksection  was  revised  to  contain  the 

following  question:  "What  is  your  opinion  of 
the  reviewed  products  ,  based  on  the  data 
elements  (or  metrics  or  criteria)  above?" 

(1-10)  -  (0  if  you  have  no  opinion).  More 
detail  is  provided  in  this  Theoretical 
Supplement  in  Section  V. 

There  should  be  information  provided  explaining 
the  concept  of  threshold  values  based  on  historical 
data. 

#17:  This  information  is  found  in  this  Theoretical 

Supplement  ir.  Section  V  and  in  Section  JV  of  the 
General  Instructions . 


i 

i 
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SECTION  V 


CONCLUSIONS  AND  RECOMMENDATIONS  FOR  FURTHER  ACTION 

5.1  ALTERNATIVE  USES  FOR  SOFTWARE  METRICS 

Software  Metrics  is  a  set  of  procedures  for  measuring  es¬ 
sential  aspects  of  software  systems.  Software  Metrics  was 
principally  designed  to  be  used  as  a  tool  to  measure  certain 
qualities  in  a  system  under  development  through  the  calculation 
of  Quality  Factor  scores  which  are  ultimately  evaluated  against 
historical  data.  These  measures  of  the  Quality  Factor  of  a  sys¬ 
tem  are  to  be  used  as  "early-warning”  signals  to  point  out  de¬ 
ficiencies  as  they  develop,  so  that  they  can  be  corrected  as 
easily  and  inexpensively  as  possible.  However,  it  is  not  always 
possible  to  use  Software  Metrics  in  this  way  because  of  the 
changes  in  policy  or  unforseen  events  during  a  software  develop¬ 
ment.  Therefore,  versatility  was  designed  into  Software  Metrics 
in  order  to  allow  alternative  utilizations  of  this  tool. 

SAI  has  identified  six  alternative  methods  of  using  Soft¬ 
ware  Metrics;  (1)  Use  as  measure  of  the  impact  of  revisions  to 
software,  (2)  Use  as  a  review  tool  to  determine  the  appropriate¬ 
ness  of  redesign,  (3)  Use  to  perform  retrospective  analyses  of 
existing  systems,  (4)  Use  as  performance  incentives  based  on 
Software  Metrics  scores,  (5)  Use  to  develop  guidelines  for  in- 
house  development  of  software,  and  (6)  Use  to  maintain  control 
and  visibility  of  software  development.  Each  of  these  alter¬ 
natives  will  be  discussed  further  in  the  remainder  of  this  section. 

5.1.1  Software  Metrics  Used  to  Measure  Software  Revisions 

When  revisions  are  made  to  software,  the  impact  of 
these  revisions  could  be  measured  by  applying  Software 
Metrics  to  the  revised  software.  If  the  same  Quality 


Factors  are  chosen  for  the  revised  application  and  scores 
of  the  original  software  have  been  maintained,  the  revised 
and  original  scores  could  be  compared  to  determine  if  the 
revisions  had  any  negative  or  positive  effect  on  the  soft¬ 
ware.  However,  if  different  qualities  have  been  determined 
to  be  important,  separate  sets  of  Quality  Factors  could  be 
applied  to  the  revised  software.  In  this  case,  these  scores 
could  be  maintained  as  historical  data  for  future  reference 
and  comparison. 

5.1.2  Software  Metrics  Used  as  a  Review  Tool 

When  a  redesign  of  some  software  is  being  considered. 
Software  Metrics  could  be  applied  to  the  existing  software 
to  determine  the  weak  areas  in  the  design  with  respect  to 
the  newly  desired  characteristics.  The  redesign  could  be 
guided  by  the  Quality  Factor  scores  that  are  low  with  res¬ 
pect  to  the  newly  desired  characteristics.  For  instance, 
if  a  particular  piece  of  software  was  not  originally  written 
with  maintainability  in  mind,  the  Maintainability  Quality 
Factor  Module  could  be  applied  to  the  software.  If  Main¬ 
tainability  scored  low,  then  work  to  improve  maintainability 
would  be  needed. 

5.1.3  Software  Metrics  Used  to  Perform  Retrospective  Analyses 

For  systems  that  were  developed  without  the  Metrics 
being  applied  during  development.  Software  Metrics  could  be 
applied  after  development  is  over  to  determine  where  in  the 
development  problems  if  any  originated.  For  example,  if  it 
has  been  determined  that  a  particular  system  is  very  difficult 
to  maintain,  the  Maintainability  Software  Metrics  could  be 
applied  to  all  of  the  development  documentation  to  find  the 
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source  of  the  maintainability  problem.  Alternatively,  if  a 
system  is  determined  to  be  outstanding  in  a  particular 
quality,  the  Software  Metrics  could  be  applied  to  that  soft¬ 
ware  and  the  results  could  be  used  as  examples  of  good 
scores  to  compare  against  future  development  efforts. 

5.1.4  Software  Metrics  Used  as  Performance  Incentives 

Software  contractors  could  be  given  performance  in¬ 
centives  based  on  Software  Metric  scores.  Software  would 
be  evaluated  based  on  Software  Metric  scores.  This  would 
provide  incentive  to  the  contractor  to  design  the  desired 
qualities  into  the  software. 

5.1.5  Software  Metrics  Used  to  Develop  Guidelines 

If  measurement  of  software  quality  is  not  desired,  or 
if  historical  figures  are  not  yet  available  so  that  that 
type  of  evaluation  is  not  possible,  software  could  be  designed 
and  implemented  following  guidelines  developed  from  the  Data 
Elements  of  the  Metrics  Handbook.  Instead  of  measuring  the 
existence  of  Quality  Factors,  the  Data  Element  questions 
can  be  used  to  develop  guidelines  that  reflect  good  pro¬ 
gramming  practice. 

5.1.6  Software  Metrics  Used  for  Control  and  Visibility 

Software  Metrics  provides  a  stuctured  approach  to 
control  and  observation  of  the  development  of  software 
throughout  its  life  cycle.  If  the  software  metric  data  is 
collected,  but  not  calculated  to  produce  metric  scores,  it 
provides  control  and  visibility  to  the  development.  Soft¬ 
ware  Metrics  data  tracks  the  development  of  the  system  via 
the  products  produced  during  the  life  cycle.  Life  Cycle 
control  of  the  development  is  thus  easily  accomplished. 
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5 . 2  MANAGEMENT  CONCERNS 


The  use  of  Software  Metrics  as  a  Quality  Assurance  tool  re¬ 
quires  continuing  concern  by  management  of  two  major  issues.  The 
first  issue  is  the  granularity  and  subjectivity  of  several  of  the 
Data  Element  questions.  The  second  issue  is  the  necessity  to 
track  and  monitor  the  collection  of  scores  and  evaluations  and 
the  developments  of  a  Metric  Data  Base. 

Granularity  and  subjectivity  of  the  Data  elements  can  be 
decreased  over  time  by  developing  standard  guidelines  for  re¬ 
solving  each  of  the  questions  which  are  determined  to  lack  granu¬ 
larity  or  are  too  subjective.  A  procedure  for  monitoring  and  re¬ 
solving  these  issues  should  be  developed. 

The  second  issue  that  management  must  consider  is  the  neces¬ 
sity  to  track  and  monitor  both  the  collection  of  the  scores  and 
evaluations.  Tracking  and  monitoring  of  the  scores  is  necessary 
in  order  to  develop  a  Metric  data  base.  These  scores  can  be  used 
to  compare  future  re -applications  of  Software  Metrics  to  the  same 
system,  or  in  the  devleopment  of  other  systems.  The  data  base  can 
be  used  to  compare  history  with  current  development  efforts. 

Each  Metric,  Criteria,  and  Factor  worksheet  contains  an 
Evaluation  Worksection.  It  is  important  to  note  that  this  eval¬ 
uation  is  subjective  on  the  part  of  the  person  applying  the  Soft¬ 
ware  Metrics.  For  instance,  at  the  Criteria  level,  the  Evaluation 
Worksection  asks:  "What  is  your  evaluation  of  the  reviewed  pro¬ 
ducts  based  on  the  Metrics  above?  _ (1-10  or  0  if  you  are 

unable  to  evaluate)."  When  a  person  is  evaluating  a  Criteria  it 
would  be  possible  to  have  one  Metric  with  a  high  score  and  another 
with  a  low  score.  The  evaluator  could  decide  that  the  Criteria 
should  be  evaluated  fairly  high  (5-8)  because  in  his  opinion  the 
low  scoring  Metric  did  not  have  much  of  an  impact  on  the  system. 


Because  of  this  subjective  nature,  a  particular  evaluator  may 
consistantly  evaluate  high,  or  consistently  evaluate  low.  Some 
method  of  tracking  and  monitoring  this  type  of  scoring  should  be 
developed  so  that  an  analysis  of  the  scores  and  scorer  can  be 
done  as  a  means  of  giving  a  proper  interpretation  to  the  histori¬ 
cal  data  in  the  data  base. 

S . 3  CONCLUSIONS  AND  RECOMMENDATIONS 

Software  Metrics  is  a  state-of-the-art  tool,  and  as  such 
is  still  in  its  infancy.  Therefore,  some  of  the  objectivity  de¬ 
sired  by  a  quantification  of  software  quality  has  not  yet  devel¬ 
oped.  Objectivity  develops  as  guidelines  are  set  and  as  threshold 
values  are  developed.  Secondly  Software  Metrics  are  immediately 
valuable  in  their  present  subjective  state  because  it  can  be  used 
as  a  checklist  for  monitoring  software  throughout  its  acquisition 
life  cycle.  This  makes  Software  Metrics  easy  to  use,  easy  to 
teach,  and  not  time  consuming  to  apply.  This  is  currently  the 
most  feasible  use  of  the  software  metrics.  Use  as  a  guideline 
and  for  control  and  visibility  as  earlier  discussed  in  5.1.5  and 
5.1.6,  provide  immediate  value  from  their  application. 

SAI  recommends  four  actions  be  taken  by  ESD  to  build  a 
metric  history:  (1)  Apply  the  Software  Metrics  to  a  wide  variety 
of  software  development  efforts.  As  systems  reach  the  maintain- 
ence  phase  independent  assessments  of  the  quality  of  the  system 
should  be  performed  and  compared  to  the  quality  evaluation  as 
measured  by  metrics;  (2)  Establish  a  process  to  evaluate  and  up¬ 
date  threshold  values  as  they  are  developed;  (3)  Maintain  a  Soft¬ 
ware  Metrics  historical  data  base;  and  (4)  Improve  threshold 
value  believability  over  time  by  comparing  independent  quality 
assessments  with  the  Software  Metric  Data  Base. 
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